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(54) Communication terminal device with tactile feedback 



(57) A sending communication terminal device in- 
cludes a physical quantity detection sensor for detecting 
the physical quantities that affect the sending commu- 
nication terminal device, and transmits the physical 
quantities as physical quantity data in a form to be dis- 
tinguished from other types of data. The receiving com- 



munication terminal device extracts the physical quan- 
tity data from received data, analyzes the physical quan- 
tities, processes other kinds of received data or data that 
has been prepared in the receiving communication ter- 
minal device according to the analysis result, and out- 
puts the processed data on the display. 
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Description 

BACKGROUND OF THE INVENTION 

(1 )Field of the Invention s 

The present invention relates to a communication 
terminal device that transmits and receives data, and 
especially relates to a communication terminal device 
that transmits and receives physical quantities and im- 
age manipulations which affect the sending communi- 
cation terminal device. 

(2)Related Art 

It is possible to transmit and receive audio data, im- 
age data, text data, animation data, or the like in its orig- 
inal form using a conventional communication terminal 
device or a communication system, such as a tele- 
phone, facsimile, pager, computer communication sys- 
tem or the like. 

With the recent growing demand for portable com- 
munication terminal devices and the recent develop- 
ments in computer technology, it is possible for a port- 
able computer terminal device to transmit and receive 
various kinds of data. In other words, a portable compu- 
ter terminal device can achieve multimedia communica- 
tion. 

Conventional communication terminal devices 
have concentrated on the minimization of errors when 
mechanically transmitting information that represents 
the data that is to be transmitted to another communi- 
cation terminal device. 

As a result such conventional communication ter- 
minal devices are not able to transmit an action of the 
user of a sending communication terminal device and 
to have the user of the receiving communication termi- 
nal device act in response to the action of the sending 
communication terminal device user, for instance. 

Actions of the user of the sending communication 
terminal device, however, can be a means to express 
an emotion of the sending communication terminal de- 
vice user. When an action of the sending communication 
terminal device user can be transmitted to the receiving 
communication terminal device to have the receiving 
communication terminal device user act in response to 
the action, the communication between the sending 
communication terminal device user and the receiving 
communication terminal device user may be diversified. 

SUMMARY OF THE INVENTION 

An object of the present invention is to provide a 
communication terminal device that efficiently diversi- 
fies the communication between the sending communi- 
cation terminal device user and the receiving communi- 
cation terminal device user. 

Another object of the present invention is to provide 
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a new communication terminal device that conveys the 
actions of the sending communication terminal device 
user reflecting his/her emotions in the manners that ap- 
peal to the senses, especially to the visual sense, of the 
receiving communication terminal device user. 

A further object of the present invention is to provide 
a communication terminal device that converts physical 
actions taken by the sending communication terminal 
device user to non-physical ones to be displayed to the 
receiving communication terminal device user. 

Yet another object of the present invention is to pro- 
vide a communication terminal device available for the 
two-way communication in which two communication 
terminal devices transmit the actions taken by the users 
to each other. 

A still further object of the present invention is to 
provide a communication terminal device that transmits 
the information reflecting the actions of the sending 
communication terminal device user and the receiving 
communication terminal device user without requiring 
large amount of transmission capacity. 

An additional object of the present invention is to 
provide a communication system where the actions tak- 
en by the sending communication terminal device user 
is transmitted to and displayed on the receiving commu- 
nication terminal device. 

The above-described objects are realized by the 
combination of the sending communication terminal de- 
vice and the receiving communication terminal device 
described below. 

The sending communication terminal device is a 
sending communication terminal device that transmits 
data to a receiving communication terminal device, in- 
cluding: a detection unit for detecting physical quantities 
that affect one of a main body of the sending communi- 
cation terminal device and a part of the sending com- 
munication terminal device; a preparation unit for pre- 
paring one of target data that is to be processed with the 
physical quantities detected by the detection unit and 
an identifier for identifying the target data; and a trans- 
mission unit for transmitting the physical quantities and 
one of the target data and the identifier prepared by the 
preparation unit in a form that enables the physical 
quantities and one of the target data and the identifier 
to be identified. 

The sending communication terminal device may 
be the sending communication terminal device, wherein 
the transmission unit includes a link information addition 
unit for adding link information, which relates the phys- 
ical quantities with the target data, to at least the phys- 
ical quantities. 

The sending communication terminal device may 
be the sending communication terminal device, wherein 
the detection unit includes: an action detection sensor 
for detecting an action that is performed by a user of the 
sending communication terminal device on one of a 
main body of the sending communication terminal de- 
vice and a part of the sending communication terminal 
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device; and a physical quantity data output unit for cre- 
ating data relating to an action type and a direction using 
signals detected by the action detection sensor and for 
outputting the created data. 

The sending communication terminal device may 
be the sending communication terminal device further 
including a display for displaying an image that is the 
target data that is to be processed with the physical 
quantities, wherein the detection unit includes a touch 
panel that is provided on the display, and a processing 
circuit that detects an action performed by the user of 
the sending communication terminal device on the 
touch panel and interprets the detected action per- 
formed by the user as one of pressing, stroking, and rub- 
bing. 

The sending communication terminal device may 
be the sending communication terminal device, wherein 
the target data is symbol image data that metaphorically 
represents a user of the sending communication termi- 
nal device, and the preparation unit includes a storage 
unit for storing the symbol image data. 

The sending communication terminal device may 
be the sending communication terminal device, wherein 
the symbol image data is three-dimensional image data, 
and the transmission unit performs a process by which 
the three-dimensional image data is transmitted at a be- 
ginning of communication with the receiving communi- 
cation terminal device. 

The receiving communication terminal device is a 
receiving communication terminal device that receives 
data transmitted from a sending communication termi- 
nal device, including: a reception unit for receiving the 
data transmitted from the sending communication ter- 
minal device; a division unit for grouping the data trans- 
mitted from the sending communication terminal device 
into a first data that represents physical quantities de- 
tected by the sending communication terminal device 
and a second data that is to be processed with the phys- 
ical quantities; a processing unit for processing the sec- 
ond data according to the first data; and an expressing 
unit for expressing for a user of the receiving communi- 
cation terminal device the second data after the 
processing by the processing unit. 

The receiving communication terminal device may 
be the receiving communication terminal device, where- 
in the division unit includes a data type judgement unit 
for detecting identifiers that are added to the data trans- 
mitted from the sending communication terminal device, 
and for distinguishing the first data and the second data 
based on the detected identifiers. 

The receiving communication terminal device may 
be the receiving communication terminal device, where- 
in the processing unit includes: an analysis unit for an- 
alyzing the first data that affects the sending communi- 
cation terminal device to detect a type of the physical 
quantities; and a processing unit for processing the sec- 
ond data according to an analysis result of the analysis 
unit 
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The receiving communication terminal device may 
be the receiving communication terminal device, where- 
in the second data obtained from the grouping by the 
division unit includes audio data and image data, the au- 

5 dio data is voice of a user of the sending communication 
terminal device, the image data is symbol image data 
that metaphorically represents the user of the sending 
communication terminal device, the analysis unit ana- 
lyzes the first data to determine that the physical quan- 
go tities that affect the sending communication terminal de- 
vice are for a vibration and or a swing, the processing 
unit refers to an analysis result of the analysis unit, adds 
vibrato to the audio data, and shakes a head of the sym- 
bol image data of the sending communication terminal 

is device user in one of vertical and horizontal directions, 
according to the result and the expressing unit includes 
a loudspeaker that emits the audio data after processing 
by the processing unit and a display that displays the 
image data after processing by the processing unit. 

20 The receiving communication terminal device may 
be the receiving communication terminal device, where- 
in the symbol image data of the sending communication 
terminal device user is three-dimensional image data, 
the processing unit includes: an image data creation unit 

25 for creating variations of the image data seen from a plu- 
rality of camera angles using the three-dimensional im- 
age data; and a selection unit for selectively outputting 
the variations of the image data one at a time to the dis- 
play according to the analysis result of the analysis unit. 

30 The receiving communication terminal device may 
be a receiving communication terminal device that re- 
ceives data transmitted from a sending communication 
terminal device including: a reception unit for receiving 
the data transmitted from the sending communication 

35 terminal device; a data type judgement unit for detecting 
identifiers that are added to the data transmitted from 
the sending communication terminal device, and for dis- 
tinguishing first data that represents physical quantities 
and second data that is other data, based on the detect- 

40 ed identifiers; a to-be-processed data extraction unit for 
detecting link information included in at least one of the 
first data and the second data, and extracting data, from 
the second data, which is related to the first data by the 
link information; a processing unit for processing the ex- 

45 tracted data with the first data; and an expressing unit 
for expressing for a user of the receiving communication 
terminal device the extracted data after processing by 
the processing unit. 

The receiving communication terminal device may 

50 be a receiving communication terminal device that re- 
ceives data transmitted from a sending communication 
terminal device including: a reception unit for receiving 
the data transmitted from the sending communication 
terminal device; a division unit for grouping the data 

ss transmitted from the sending communication terminal 
device into a first data that represents physical quanti- 
ties detected by the sending communication terminal 
device and a second data that specifies data that is to 
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be processed with the physical quantities; a storage unit 
for storing the data to be processed; a processing unit 
for extracting the data that is to be processed and is 
specified by the second data, and for processing the ex- 
tracted data according to the first data; and an express- 
ing unit for expressing for a user of the receiving com- 
munication terminal device the extracted data after the 
processing by the processing unit. 

The above-described objects are also realized by a 
communication terminal device that performs two-way 
communication with another communication terminal 
device including: a display; a physical quantity detection 
unit for detecting an action performed on a first image 
that is displayed on the display to create first operation 
data, and for detecting an action performed on a second 
image that is displayed on the display to create second 
operation data; a preparation unit for preparing at least 
one of image data and audio data corresponding to at 
least one of the first and the second images; a transmis- 
sion unit for transmitting the first operation data, the sec- 
ond operation data, and a preparation data in a form 
which enables each of the first operation data, the sec- 
ond operation data, and the preparation data to be iden- 
tified; a reception un it for receiving data transmitted from 
the other communication terminal device; a data judge- 
ment unit for judging that the data transmitted from the 
other communication terminal device is one of received 
first operation data, received second operation data, 
and received preparation data; a first processing unit for 
processing the received preparation data that is re- 
ceived by the reception unit according to the received 
first operation data; a second processing unit for 
processing the received preparation data that is re- 
ceived by the reception unit according to the received 
second operation data; an activation unit for selectively 
activating one of the first processing unit and the second 
processing unit according to a result obtained by the da- 
ta type judgement unit; a display control unit for having, 
when the preparation data that is processed by one of 
the first processing unit and the second processing unit 
is image data, a corresponding image on the display re- 
placed with contents of the preparation data; and a 
sound emission unit for emitting the contents of the 
preparation data when the preparation data that is proc- 
essed by one of the driven first processing unit and the 
second processing unit is audio data. 

The communication terminal device may be the 
communication terminal device, wherein the first image 
represents a user of the communication terminal device, 
and the second image represents a user of the other 
communication terminal device. 

The communication terminal device may be the 
communication terminal device, wherein the image data 
prepared by the preparation unit is three-dimensional 
image data of a symbol image that metaphorically rep- 
resents the user of the communication terminal device. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects, advantages and features 
of the invention will become apparent from the following 
5 description thereof taken in conjunction with the accom- 
panying drawings which illustrate a specific embodi- 
ment of the invention. In the Drawings: 

Fig. 1 is a perspective drawing of the portable com- 
munication terminal device according to the first 
embodiment of the present invention; 
Fig. 2 shows a hardware construction of the porta- 
ble communication terminal device according to the 
first embodiment of the present invention; 
Fig. 3 is a functional block diagram of the portable 
communication terminal device according to the 
first embodiment of the present invention; 
Fig. 4A is an example of the images shown on the 
display of the sending portable communication ter- 
minal device; 

Fig. 4B is an example of the images shown on the 
display of the receiving portable communication ter- 
minal device; 

Fig. 5 shows the data structure of information data 
that the communication terminal devices transmit 
and receive; 

Fig. 6 shows the data structure of physical quanti- 
ties data that the communication terminal devices 
transmit and receive; 

Fig. 7 shows the data structure of symbol image da- 
ta that the communication terminal devices transmit 
and receive; 

Fig. 8 is a sequence diagram showing the proce- 
dure in which communication is performed between 
the communication terminal devices; 
Fig. 9 is a flowchart that shows the first half of the 
transmission control process by the communication 
terminal device; 

Fig. 10 is a flowchart that shows the second half of 
the transmission control process by the communi- 
cation terminal device; 

Figs. 11 to 1 3 are flowcharts that show the action 
control process by the communication terminal de- 
vice; 

Fig. 14 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the sending communication ter- 
minal device is vibrated; 

Fig. 15 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the sending communication ter- 
minal device is swung in the horizontal direction; 
Fig. 16 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the sending communication ter- 
minal device swings in the vertical direction; 
Fig. 17 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
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nal devices when pressure is applied to the sending 
communication terminal device; 
Fig. 1 8 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the symbol image of the receiving s 
communication terminal device user displayed on 
the sending communication terminal device is 
pressed; 

Fig. 19 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the symbol image of the receiving 
communication terminal device user displayed on 
the sending communication terminal device is 
stroked; 

Fig. 20 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the height of the symbol image of 
the sending communication terminal device user 
displayed on the sending communication terminal 
device is increased; 

Fig. 21 shows a typical example of the correspond- 
ing actions performed by the communication termi- 
nal devices when the height of the symbol image of 
the sending communication terminal device user 
displayed on the sending communication terminal 
device is decreased; 

Fig. 22A shows the actions by the receiving com- 
munication terminal device when the operation unit 
operates in the predetermined order in the "consec- 
utive operation mode". 

Fig. 22B shows the actions by the receiving com- 
munication terminal device when the operation unit 
operates in another predetermined order in the 
'consecutive operation mode". 
Fig. 23A is a functional block diagram of the sending 
communication terminal device according to the 
second embodiment of the present invention; 
Fig. 23B is a functional block diagram of the receiv- 
ing communication terminal device according to the 
second embodiment of the present invention; 
Fig. 24 shows the process performed by the send- 
ing and the receiving communication terminal de- 
vices of Figs. 23A and 23B; 
Fig. 25 is a functional block diagram of the sending 
communication terminal device according to the 
third embodiment of the present invention; and 
Fig. 26 shows the data structure of the instruction 
data that the communication terminal devices trans- 
mit and receive, 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The following is the explanation of the first embod- 
iment of the present invention with reference to the fig- 
ures. Fig. 1 is a perspective drawing of portable com- 
munication terminal device A according to the present 
embodiment of the invention. Fig. 2 is a hardware con- 
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figuration of portable communication terminal device A. 
Portable communication terminal device A includes 
central processor 100, pen-input tablet 101a, micro- 
phone 101b, acceleration sensor 101c, pressure sen- 
sors 101 d, display 102a, loudspeaker 102b, drive 102c, 
LEDs (light emitting diode) 102d, communication unit 
103, and IC (integrated circuit) card reader 104a as its 
main components. 

Central processor 100 includes CPU (central 
processing unit) 100a for performing operations, RAM 
(random access memory) 1 00b that is a work storage, 
and ROM (read only memory) 100c for storing the pro- 
gram for data processing and communication. 

Pen-input tablet 101a, microphone 101b, accelera- 
tion sensor 1 01 c, and pressure sensors 1 01 d are includ- 
ed in input unit 1 01 . Pen-input tablet 1 01 a is formed over 
display 102a. As a result, when one point on pen-input 
tablet 101a is pressed with pen 101a', the coordinates 
of this point on display 1 02a may be input in central proc- 
essor 100. More specifically, when a character or a fig- 
ure is input using pen-input tablet 101 a and central proc- 
essor 100, the character or the figure may be decoded. 
When pen 101a' is used to press pen-input tablet 101a 
or is dragged across pen-input tablet 1 01 a, this physical 
action by the user may be detected as "stroke data". In- 
structions about an action such as increasing or de- 
creasing the height of the image on display 102a are 
also input. Microphone 101b transforms audio data into 
electrical signals. The audio data is digitized in a follow- 
ing process. 

Acceleration sensor 101c is, for instance, a servo- 
type sensor or an electrostatic-capacity-type sensor. 
Two acceleration sensors 101c(x) and 101c(y) are in- 
cluded in portable communication terminal device A as 
shown in Fig. 1 , and detect the acceleration of portable 
communication terminal device A in the X-axis direction 
and in the Y-axis direction respectively. 

Pressure sensors 101d, which can be one of vari- 
ous kinds of sensor, detects pressure by detecting the 
pressure applied to portable communication terminal 
device A using a distortion gauge in this specification. 
Pressure sensors 101d are placed at the positions at 
which pressure is applied by the user's hands when the 
user holds portable communication terminal device A in 
the hands. 

Input unit 1 01 that is described above can of course 
include other input means, such as a keyboard, a scan- 
ner, and a camera. 

Display 102a, loudspeaker 102b, drive 102c, and 
LEDs 1 02d are included in output unit 1 02. Display 1 02a 
displays an image, text, or a menu. Loudspeaker 1 02b 
outputs audio data. Drive 102c vibrates when driven by 
a motor and vibrates portable communication terminal 
device A. LEDs 102d are formed of adjacent arrays of 
red and green LEDs so that different colors may be dis- 
played as a combination of red and green. Component 
102d' that covers LEDs 102d is translucent to allow the 
light emitted from LEDs 102d to pass. 
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Microphone 101b and loudspeaker 102b are built 
into the main body of portable communication terminal 
device A, and are also provided on headset 200. Head- 
set 200 is connected to portable communication termi- 
nal device A by a cord. The user wearing headset 200 
may listen to audio data output from loudspeaker 102b 
or input audio data with microphone 101b, while holding 
portable communication terminal device A in the hands 
and watching display 1 02a or operating pen-input tablet 
101a. 

Communication unit 103 converts data into the sig- 
nals to be transmitted, and includes modem 103a for 
converting transmitted signals to be processed by cen- 
tral processor 100, high frequency unit 103b for 
processing signals to be transmitted and received, and 
antenna 103c for transmitting and receiving signals. IC 
card reader 104a reads data from IC card 104b that 
stores data including programs, and transmits the data 
to central processor 100. 

Portable communication terminal device A may be 
functionally divided into transmission system 1 and re- 
ception system 2. A functional block diagram of trans- 
mission system 1 and reception system 2 is shown in 
Fig. 3. 

Mode switching direction unit 31 in Fig. 3 functions 
in both transmission system 1 and reception system 2. 
Mode switching direction unit 31 gives indication for 
switching the present mode of portable communication 
terminal device A and the other communication terminal 
device that maintains communication with portable 
communication terminal device A. In the present exam- 
ple, "consecutive operation mode", "regular mode", and 
"video, still picture, or text display mode" are the avail- 
able modes. "Consecutive operation mode" is a mode 
that will be described later where a series of operations 
is consecutively performed by the receiving communi- 
cation terminal device as shown in Fig. 22. "Regular 
mode" is a mode in which symbol images that represent 
the users of the two communication terminal devices are 
exchanged and audio communication is mainly per- 
formed, with the symbol images being displayed on 
each of the displays as shown in Fig. 4. "Video, still pic- 
ture, and text display mode" is a mode in which symbol 
images are not displayed even if an image has been 
transmitted from the other communication terminal de- 
vice or has already been stored in the memory, and in 
which animation data, still picture data, or text data that 
is transmitted from the other communication terminal 
device is displayed. The mode is switched using the 
mode switch screen that appears on the display when 
the user presses a predetermined point on pen-input 
tablet 1 01 a. Mode switch instructions are to be transmit- 
ted from transmission system 1 to the other communi- 
cation terminal device as instruction data. 

Transmission system 1 includes information data 
input unit 11, physical quantity detection unit 12, physi- 
cal quantity data output unit 1 3, image manipulation unit 
14, image manipulation data output unit 15, input data 



storage unit 16, transmission control unit 17, transmis- 
sion data instruction unit 18, transmission unit 19, and 
window 20a. 

Information data input unit 11 accepts multimedia 
s data input. More specifically, information data input unit 
11 accepts audio data, text data, still picture data, ani- 
mation data, and symbol images that metaphorically 
represent the users. Audio data is input with microphone 
101b. Text data, still picture data, animation data, and 
symbol image data is input with pen-input tablet 101a 
or IC card reader 104a. In this specification symbol im- 
age data is three-dimensional image data to enable the 
receiving communication terminal device to easily cre- 
ate and show wide variations of symbol images using 
the transmitted symbol image data, and to minimize the 
amount of data transmitted to the receiving communi- 
cation terminal device. 

Physical quantity detection unit 12 detects physical 
quantities that affect portable communication terminal 
device A, and includes acceleration sensor 101c for 
measuring the acceleration in the vertical and horizontal 
directions for entire portable communication terminal 
device A, and pressure sensors 1 01d for measuring the 
pressure applied to portable communication terminal 
device A. Other kinds of physical quantities, such as ro- 
tation frequencies and angular acceleration, impact that 
portable communication terminal device A receives, 
temperature, illuminance, and atmospheric pressure 
may be detected with a variety of well-known sensors. 

Physical quantity data output unit 1 3 outputs the in- 
formation of the detected physical quantities as physical 
quantity data based on the detection result of physical 
quantity detection unit 12. More specifically, physical 
quantity data output unit 1 3 analyzes parameters, cal- 
culates the numerical value of predetermined physical 
quantities, and digitizes the numerical values to be proc- 
essed by CPU 100a. In this specification, physical quan- 
tity data output unit 1 3 integrates the detection result by 
acceleration sensor 101c to calculate the moving 
speeds of portable communication terminal device A in 
the vertical and horizontal directions, calculates the 
amount of pressure applied to portable communication 
terminal device A using the detection result of pressure 
sensors 101d, digitizes the moving speeds and the 
amount of pressure, and inputs the digitized data into 
input data storage unit 16. 

Window 20a shows the image on which actions are 
performed in image manipulation unit 14, and appears 
in a predetermined area on display 102. While the im- 
ages that represent the users of the sending and receiv- 
ing communication terminal devices are symbol images 
in this specification (refer to Fig. 4), other images such 
as text images may be used. The symbol image that rep- 
resents the user of the communication terminal device 
itself is created using the data input by information data 
input unit 11. The symbol image shown on the display 
or the window is a front view of a three-dimensional im- 
age with a camera set in front of the image. The symbol 
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image of the other communication terminal device user 
is transmitted to the communication terminal device at 
the beginning of the communication, more specifically, 
just before the connection, and is shown on the window 
of the communication terminal device. The image 
shown on the window of the communication terminal de- 
vice is edited by the image editing unit (described later) 
in reception system 2, and shows variations according 
to the physical quantity data transmitted from the other 
communication terminal device. It is of course possible 
for the communication terminal device to store the sym- 
bol images of the users of the other communication ter- 
minal devices and the communication terminal device 
itself, so that when these communication terminal de- 
vices start the communication, it is possible to detect the 
symbol image of the other communication terminal de- 
vice user using the identifier of the other communication 
terminal device and to show the symbol image on the 
window of the communication terminal device. 

Image manipulation unit 14 performs certain ac- 
tions on the image shown on window 20a. The certain 
actions here include pressing and stroking the symbol 
image of the receiving communication terminal device 
user by pressing and stroking pen-input tablet 101a, 
which is provided on display 102a, with the point of pen 
101a 1 , and to move the point of pen 101a' on the surface 
of pen-input tablet 101a up or down with pressing the 
head of the image. When the user moves the point of 
pen 101a' up with pressing the head of the image with 
the point of the pen, central processor 1 00 interprets the 
move as the input of instructions. As a result, the height 
of the symbol image shown on window 20a increases. 
When the user moves the point of pen 101a' down with 
pressing the head of the image with the point of the pen, 
the height of the symbol image shown on window 20a 
decreases in the same manner. Such image processing 
in which the height of the symbol image changes is per- 
formed in image manipulation unit 1 4 into which the data 
of the symbol image is input from information data input 
unit 11. The image processing is realized using well- 
known techniques. The following is an explanation of 
one example of such an image processing method. 
First, two-dimensional image data is extracted from the 
data of the three-dimensional symbol image, and the ra- 
tio of the Y coordinate of each point of the two-dimen- 
sional image data to the Y coordinate of the initial point 
of the point of pen 101a' is calculated. Then the amount 
of shift of the Y coordinate of each point is obtained by 
multiplying the calculated ratio of each point by the 
amount of shift of pen 101a' moved from the initial point, 
and the obtained amount of shift is added to the value 
of Y coordinate of each point. As a result, the height of 
the image may increase or decrease. Other actions may 
be performed on the image shown on window 20a. For 
instance, another image may be added to the image, 
part of the image may be deleted, or the image may be 
replaced by a newly input image. 

Image manipulation data output unit 15 outputs the 



information about the action performed on the image in 
image manipulation unit 1 4 as image manipulation data. 
More specifically, when a predetermined action is per- 
formed on the image in image manipulation unit 14, im- 
s age manipulation data output unit 15 outputs the code 
sequence corresponding to the action as image manip- 
ulation data. 

For instance, code "00" represents an action in 
which the sending communication terminal device user 
presses the image of the receiving communication ter- 
minal device user with the point of pen 101a', and code 
"01 M represents an action in which the sending commu- 
nication terminal device user strokes the image of the 
receiving communication terminal device user with the 
point of pen 101a'. Code "10" represents an action in 
which the sending communication terminal device user 
presses the head of the image of the sending commu- 
nication terminal device user with the point of pen 1 01a' 
and moves the point of the pen up, and code "11" rep- 
resents the action in which the sending communication 
terminal device user presses the head of the image of 
the sending communication terminal device user with 
the point of pen 101a' and moves the point of the pen 
down. A header that indicates that the data is image ma- 
nipulation data is added to each piece of the code, and 
the code with the header is output as image manipula- 
tion data. In this process when a certain period of time 
has passed after the completion of one of the above- 
mentioned image manipulations by the sending commu- 
nication terminal device user, image manipulation data 
output unit 1 5 outputs a reset signal to inform the receiv- 
ing communication terminal device user of the comple- 
tion of the image manipulation. Such image manipula- 
tion data is detected when a physical action is performed 
on portable communication terminal device A, that is, 
when pen-input tablet 101a is pressed or stroked with 
pen 1 01 a 1 , so that the image manipulation data is a kind 
of physical quantity data in a broad sense. 

Input data storage unit 16 is composed of RAM 104 
and stores the data output from information data input 
unit 1 1 , physical quantity data output unit 1 3, and image 
manipulation data output unit 1 5, in addition to the mode 
switch instruction data output from mode switch instruc- 
tion unit 31 . Figs. 5 and 6 show the data structure of the 
information data and physical quantity data stored in in- 
put data storage unit 16. The symbol image data in the 
information data is three-dimensional data and is sepa- 
rately shown in Fig. 7. As a result, the information data 
in the data structure of Fig. 5 shows information data 
apart from symbol image data. The data structure of in- 
struction data such as mode switch instructions is 
shown in Fig. 26. 

The data structure of the information data is shown 
in Fig. 5. The information data includes an "information 
data identifier 0 , an "information data length", an "infor- 
mation data type identifier", "information data", and an 
"end-of-data identifier". The "information data identifier" 
is the identifier by which information data is distin- 
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guished trom other kinds of data such as physical quan- 
tity data and symbol image data. The "information data 
length" shows the data length of the "information data". 
The data amount of the "information data" is equal to 
the amount of data included in one transmission. For 
information data that is communicated in real time, such 
as audio data, audio data with a very short reproduction 
time is included in one transmission unit. Still picture da- 
ta and animation data is also transmitted in real time, so 
that the amount of data included in one transmission unit 
is also very small. On the other hand, in the case of sym- 
bol image data that is transmitted in one lump at the be- 
ginning of communication, the amount of data included 
in one transmission is relatively large. 

The "information data type identifier" indicates that 
the following "information data" is audio data, text data, 
animation data, or still picture data. The "end-of-data 
identifier" indicates the end of the unit of information da- 
ta. 

The data structure of the physical quantity data is 
fundamentally the same as that of the information data, 
and is shown in Fig. 6. Physical quantity data includes 
a "physical quantities identifier" for indicating that the 
data is physical quantity data, a "physical quantity data 
length" for showing the data length of the following 
"physical quantity data", a "discrete physical quantities 
identifier" for indicating the type of the "physical quantity 
data", that is, for indicating that the "physical quantity 
data" is acceleration data in the X -axis, acceleration da- 
ta in the Y-axis, pressure data, temperature data, or hu- 
midity data, "physical quantity data", and an "endof-da- 
ta identifier" for indicating the end of the unit of physical 
quantity data. The data structure of the physical quantity 
data differs from that of the information data in including 
a "physical-quantities-applied data type identifier", 
which indicates the type of information data to which the 
following "physical quantity data" is to be applied. For 
instance, when the "physical quantity data" is accelera- 
tion data, "the symbol image of the sending communi- 
cation terminal device user" is indicated by the "physi- 
cal-quantities-applied data type identifier". As a result, 
the reception system in the receiving communication 
terminal device processes the symbol image of the 
sending communication terminal device user and shows 
the symbol image that shakes or nods its head on the 
display of the receiving communication terminal device 
(refer to Figs. 15 and 16). The data structure of image 
manipulation data, which is described above, is the 
same as that of physical quantity data. 

The data structure of symbol image data is funda- 
mentally the same as that of information data. The data 
structure of symbol image data differs from that of infor- 
mation data in including a "part data length", "part data", 
and an "end-of-part-data identifier" for each component 
of a three-dimensional image and in describing each of 
the parts between "part identifier" and an "end-of-part- 
data identifier" as shown in Fig. 7. This is because sym- 
bol image data is transmitted in one lump at the begin- 



ning of communication as described above. 

The data structure of instruction data is also funda- 
mentally the same as that of information data in includ- 
ing an "instruction identifier" at the start of the structure 

5 for indicating that the data is instruction data, an "in- 
struction length" just after the "instruction identifier" for 
showing the length of the "instruction", and an "end-of- 
instruction-data identifier" in the end of the construction 
for indicating the end of the data as shown in Fig. 26. 

io The "instruction type identifier" included in the data 
structure of instruction data indicates whether the "in- 
struction" is a mode switch instruction. When the "in- 
struction" is always a mode switch instruction, the "in- 
struction type identifier" need not be included in the data 

15 structure of instruction data. 

In Fig. 3, transmission data instruction unit 18 re- 
ceives the instructions about the data to be transmitted 
and not to be transmitted from the user. The data that is 
instructed to be transmitted by the user is animation da- 

20 ta, still picture data, and text data in the information data. 
The remaining information data, which is to say, the au- 
dio data is transmitted without instructions from the user. 
The data whose transmission can be stopped is physical 
quantity data and image manipulation data. 

25 Transmission control unit 1 7 controls the order and 
the timing of data transmission when the data stored in 
input data storage unit 16 is transmitted, and in the 
present embodiment controls the transmission order of 
information data and other kinds of data in accordance 

30 with a predetermined rule. 

More specifically, when information data, physical 
quantity data, and image manipulation data are stored 
in input data storage unit 16, in principle the information 
data is transmitted first. When audio data that is a kind 

35 of information data and moving speed data that is a kind 
of physical quantity data are stored in input data storage 
unit 16, however, the physical quantity data is divided 
into a plurality of parts to be inserted between the pieces 
of the audio data without disturbing the natural flow of 

40 the sound. When audio data that is a kind of information 
data and image manipulation data including code "10" 
or "11 "are stored in input data storage unit 16, the image 
manipulation data is transmitted first and then the audio 
data is transmitted. The reset signal of image manipu- 

45 lation data is dealt with in the same way as image ma- 
nipulation data including code "00" or "01 ". 

When physical quantity data or image manipulation 
data is stored in input data storage unit 16 and informa- 
tion data is being preferentially transmitted, if the trans- 

50 mission time of the information data exceeds a prede- 
termined period of time, the data transmission of the in- 
formation data may be temporarily suspended to trans- 
mit the physical quantity data or the image manipulation 
data stored in input data storage unit 16. The interrupted 

55 information data transmission may then be resumed. 
When instruction data is stored in input data storage unit 
16, the transmission of the instruction data precedes 
any other data. 
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Transmission unit 1 9 includes communication unit 
103 and transmits the data stored in input data storage 
unit 1 6 to the other portable communication terminal de- 
vice under the control of transmission control unit 17. It 
is convenient tor the user of portable communication ter- 
minal device A to use a public network such as that used 
by the PHS (personal handyphone system) as the com- 
munication channel linking the user to other communi- 
cation terminal devices so that the user may communi- 
cate with all the subscribers of the public network. 

The following is an explanation of reception system 
2. Reception system 2 includes reception unit 21 , output 
data judgement storage unit 22, analysis unit 23, oper- 
ation control unit 24, operation unit 25, and information 
data output unit 26. 

Reception unit 21 includes communication unit 1 03 
and receives the information data, the physical quantity 
data, the image manipulation data, and the instruction 
data transmitted from the other portable communication 
terminal device. Output data judgement storage unit 22 
searches the identifier at the start of the data structure 
of the data received by reception unit 21 , judges whether 
the received data is information data, physical quantity 
data, image manipulation data, symbol image data, or 
instruction data, and temporarily stores the different 
types of received data separately. Information data or 
symbol image data is transferred to data editing unit 
25a, and physical quantity data, image manipulation da- 
ta, or instruction data is transferred to analysis unit 23. 

Analysis unit 23 analyzes the received physical 
quantity data or image manipulation data to obtain the 
physical quantities or the actions performed on the im- 
ages, and interprets the instructions in the received in- 
struction data. 

More specifically, analysis unit 23 analyzes moving 
speed data in the vertical and horizontal directions, and 
judges whether the motion of the sending communica- 
tion terminal device is reciprocating motion in the verti- 
cal or horizontal directions and whether the cycle of the 
motion is shorter than a predetermined period of time. 
When judging that the cycle of the motion is shorter than 
the predetermined period of time, analysis unit 23 judg- 
es that the sending communication terminal device is 
vibrating. When judging that the communication termi- 
nal device performs reciprocating motion in the horizon- 
tal direction and that the cycle of the motion is equal to 
or longer than the predetermined period of time, analy- 
sis unit 23 judges that the sending communication ter- 
minal device is swinging in the horizontal direction. 
When judging that the motion of the sending communi- 
cation terminal device is reciprocating motion in the ver- 
tical direction and that the cycle of the motion is equal 
to or longer than the predetermined period of time, anal- 
ysis unit 23 judges that the sending communication ter- 
minal device is swinging in the vertical direction. Analy- 
sis unit 23 analyzes pressure data and obtains the 
amount of pressure applied to the sending communica- 
tion terminal device. 



As for image manipulation data, analysis unit 23 de- 
tects which predetermined action is being performed on 
the images on the display of the sending communication 
terminal device. More specifically, analysis unit 23 de- 
$ tects one code out of codes "00" to " 1 1 " that correspond 
to the predetermined actions to be performed on the im- 
ages and are output from image manipulation data out- 
put unit 1 5. 

Operation unit 25 performs predetermined opera- 
tions, and includes data editing unit 25a, drive unit 25b, 
image editing unit 25c, illumination unit 25d, and sound 
effects unit 25e. 

Data editing unit 25a divides the information data 
transferred from output data storage unit 22, the symbol 
image data of the sending communication terminal de- 
vice user, the symbol image data of the user of the re- 
ceiving communication terminal device itself transferred 
from information data input unit 11 in transmission sys- 
tem 1 of the receiving communication terminal device 
into groups according to the type of data, transfers the 
image data of the sending communication terminal de- 
vice user and the receiving communication terminal de- 
vice user to image editing unit 25c, and edits information 
data according to the data type. For instance, when in- 
formation data is audio data, vibrato may be added to 
the voice, or the tone of the voice may be raised or low- 
ered. Still picture data and animation data is modified, 
and the font of text data is converted. The data edited 
by data editing unit 25a is transferred to information data 
output unit 26, and is shown on the display. 

Drive unit 25b realizes mechanical operations to be 
performed on the communication terminal device as a 
whole or in part. In this specification vibration of the com- 
munication terminal device performed by motor 102c is 
realized. Other actions realized by drive unit 25b may 
include bending of part of portable communication ter- 
minal device A, rotation of an antenna, and ejecting and 
retracting of a board or a stick that is free to move in and 
out of portable communication terminal device A and 
has characters or signs bearing a meaning written on it. 

Illumination unit 25d is composed of LEDs and per- 
forms operations such as changing color between red 
and green or having the LEDs flash. Sound effects unit 
25e emits audio data, such as a siren or animal sounds, 
which is stored in ROM 105 in advance and reproduced 
by CPU 100a, through speaker 102b. In this specifica- 
tion sound effects unit 25e may emit a shriek, a laugh, 
and a handclap as audio data. 

Image editing unit 25c makes and stores various 
types of images, such as an video in which the symbol 
image of the sending communication terminal device 
user shakes and nods his/her head, from the symbol im- 
age data of the users of the sending or the receiving 
communication terminal device. Image editing unit 25c 
also stores other kinds of video aside from the symbol 
images, with these being displayed when the mode of 
the communication terminal device is switched to "con- 
secutive operation mode". Image editing unit 25c may 
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edit images otherwise, and so, may change the color or 
the background of the images. 

When the mode of the communication terminal de- 
vice is switched to "consecutive operation mode", oper- 
ation unit 25 performs operations using images, sounds, 
and a light display in accordance with an order that is 
fixed in advance. More specifically, when the mode is 
switched to "consecutive operation mode", the compo- 
nents in operation unit 25 are operated so that the fol- 
lowing series of operations is repeated. The red LEDs 
in illumination unit 25d flash once, a handclap is emitted 
from sound effects unit 25e, the green LEDs in illumina- 
tion unit 25d flash once, and two handclaps are emitted 
from sound effects unit 25e, with an video in which an 
image dances to the rhythm of the flashes and the hand- 
claps being shown on the display (refer to Fig. 1 8). The 
order of the series of operations is changed by operation 
control unit 24 that will be described later. The mode is 
switched to "consecutive operation mode" by the send- 
ing communication terminal device user. 

Operation control unit 24 refers to the result of the 
analysis performed by analysis unit 23, has operation 
unit 25 perform the operations corresponding to the an- 
alyzed physical quantities, and sets operation unit 25 
according to the mode corresponding to the interpreted 
instructions. As for the physical quantities, operation 
control unit 24 has operation unit 25 perform the oper- 
ations corresponding to the result of the analysis by 
analysis unit 23 as described below. 

(1) When analysis unit 23 judges that the sending 
communication terminal device is vibrating, opera- 
tion control unit 24 has data editing unit 25a add 
vibrato to the audio data, drives drive unit 25b, and 
has image editing unit 25c output the symbol image 
of the sending communication terminal device user 
so that it is vibrating on the display (refer to Fig. 1 4). 

(2) When analysis unit 23 judges that the sending 
communication terminal device is swinging in the 
horizontal direction, operation control unit 24 has 
image editing unit 25c output the symbol image of 
the sending communication terminal device user so 
that it shakes its head on the display (refer to Fig. 
15). 

(3) When analysis unit 23 judges that the sending 
communication terminal device is swinging in the 
vertical direction, operation control unit 24 has im- 
age editing unit 25c output the symbol image of the 
sending communication terminal device user so 
that it nods its head on the display (refer to Fig. 16). 

(4) According to the amount of pressure that is ap- 
plied to the sending communication terminal de- 
vice, operation control unit 24 has illumination unit 
25d change the color of the light that is emitted from 
the LEDs. More specifically, as the amount of the 
pressure increases, the amount of the red light emit- 
ted from the LED increases, and as the amount of 
the pressure decreases, the amount of green light 



emitted from the LED increases (refer to Fig. 17). 

(5) When code "00" is detected as image manipula- 
tion data, that is, when analysis unit 23 judges that 
the symbol image of the receiving communication 

s terminal device user is pressed by the sending com- 
munication terminal device user with the point of 
pen 101a', operation control unit 24 has image ed- 
iting unit 25c show a painful expression on the sym- 
bol image of the receiving communication terminal 
device and has sound effects unit 25e emit a shriek 
(refer to Fig. 18). 

(6) When code "01 " is detected as image manipula- 
tion data, that is, when analysis unit 23 judges that 
the symbol image of the receiving communication 
terminal device user is stroked by the sending com- 
munication terminal device user with the point of 
pen 101a', operation control unit 24 has image ed- 
iting unit 25c show an amused expression on the 
symbol image of the receiving communication ter- 
minal device and has sound effects unit 25e emit a 
laugh (refer to Fig: 19). 

(7) When code "10" is detected as image manipula- 
tion data, that is, when analysis unit 23 judges that 
the sending communication terminal device user 
presses the head of the image thereof with the point 
of pen 101a' and moves the point of the pen up, op- 
eration control unit 24 raises the tone of the voice 
transmitted to data editing unit 25a (refer to Fig. 20). 

(8) When code "11 " is detected as image manipula- 
tion data, that is, when analysis unit 23 judges that 
the sending communication terminal device user 
presses the head of the image thereof with the point 
of pen 101a' and moves the point of the pen down, 
operation control unit 24 lowers the tone of the voice 
transmitted to data editing unit 25a (refer to Fig. 21 ). 

(9) When the mode is "consecutive operation mode" 
and when analysis unit 23 judges that the sending 
communication terminal device is vibrating or code 
"01" is detected as image manipulation data, oper- 
ation control unit 24 changes the predetermined op- 
erations to be performed by operation unit 25 in the 
"consecutive operation mode". More specifically, 
operation control unit 24 has the LEDs in illumina- 
tion unit 25d emit red light and green light alternate- 
ly, has sound effects unit 25e emit handclaps suc- 
cessively, and has the image in the video on display 
102a dance more vigorously (refer to Fig. 22B). 

When no physical quantity data is detected by out- 
put data judgement storage unit 22 for a certain period 
of time and a reset signal that is described above is de- 
tected by output data judgement storage unit 22, oper- 
ation control unit 24 has operation unit 25 stop perform- 
ing the operations corresponding to the physical quan- 
tity data and the image manipulation data. 

Some examples of the operations that operation 
control unit 24 has operation unit 25 perform have been 
described above. The following are other examples of 
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such actions. 

(I) When analysis unit 23 judges that the sending 
communication terminal device is swinging in the 
vertical direction, operation control unit 24 has op- 5 
eration unit 25 eject the board on which "O" is writ- 
ten. 

(I I) When physical quantity detection unit 12 in the 
sending communication terminal device detects the 
temperature of the environment, operation control 
unit 24 has operation unit 25 change the back- 
ground of the image shown on display 1 02a accord- 
ing to the temperature. 

(III) When physical quantity detection unit 12 in the 
sending communication terminal device detects the 
atmospheric pressure and the humidity of the envi- 
ronment and when analysis unit 23 judges that it is 
raining, operation control unit 24 has operation unit 
25 change the background of the image shown on 
display 102a to a picture in which it is raining. 

(IV) When physical quantity detection unit 12 in the 
sending communication terminal device detects the 
illuminance of the environment, operation control 
unit 24 has operation unit 25 change the brightness 
of the background of the image shown on display 
102a according to the illuminance. 

(V) When the phrase "I'm sorry" is input on window 
20a in image manipulation unit 14 in the sending 
communication terminal device, operation control 
unit 24 has operation unit 25 display the symbol im- 
age of the sending communication terminal device 
user that apologizes on display 102a. 

The operations described in (II) to (IV) may transmit 
not the emotions of the sending communication terminal 
device user but the environmental conditions of the 
sending communication terminal device user. In this 
sense the above-described actions may realize a great- 
er variety of communication. 

Information data output unit 26 outputs the informa- 
tion data edited by data editing unit 25a on display 1 02a 
and outputs the image data edited by image editing unit 
25c through loudspeaker 102b. When data editing unit 
25a performs no operation on information data, the in- 
formation data is output as it is. 

Transmission system 1 and reception system 2 
whose constructions have been described may be real- 
ized by installing the software that realizes the above- 
described constructions in a standard computer termi- 
nal device that includes acceleration sensor 101c, pres- 
sure sensors 1 01 d which each are included in physical 
quantity detection unit 12, motor 102c included in drive 
unit 25b, and LEDs 102d included in illumination unit 
25d. Such software may be recorded on record medium 
that is read by the above-described computer terminal 
device. Acceleration sensor 101c, pressure sensors 
101d, drive unit 25b, and illumination unit 25d may be 
included in a unit that is constructed so as to be con- 



nected to a standard computer. When such a unit is con- 
nected to a standard computer, the above-described 
software may realize the transmission system 1 and re- 
ception system 2. 

In the present embodiment the above<lescribed 
constructions are realized by recording the software on 
IC card 104b and reading the software recorded on IC 
card 104b with EC card reader 104a. The software may 
be recorded on ROM 105 in advance. 

The following is the explanation of the operations 
performed by transmission system 1 and reception sys- 
tem 2 whose constructions have been described above. 
Communication is performed between portable commu- 
nication terminal devices A1 and A2 shown in Figs. 4A 
and 4B respectively which each include transmission 
system 1 and reception system 2 whose constructions 
have been described above. Hereinafter portable com- 
munication terminal device Al will be referred to as the 
first communication terminal device, and portable com- 
munication terminal device A2 will be referred to as the 
second communication terminal device. 

When the first communication terminal device is- 
sues a call to the second communication terminal de- 
vice, the communication between the two communica- 
tion terminal devices is performed in the manner shown 
in Fig. 8. More specifically, the first communication ter- 
minal device issues the call to the second communica- 
tion terminal device, and the second communication ter- 
minal device acknowledges the receipt of the call by re- 
turning a response to the first communication terminal 
device before the line is connected. Then each of these 
communication terminal devices transmits the symbol 
image that represents the user thereof to the other com- 
munication terminal device, and transmits and receives 
audio data messages using microphones and loud- 
speakers, or other kinds of data such as physical quan- 
tity data. When the transmission of symbol images is 
completed, symbol image X2 that represents the sec- 
ond communication terminal device user and symbol 
image Y1 that represents the first communication termi- 
nal device user are displayed on the first communication 
terminal device as shown in Fig. 4A. On the other hand, 
symbol image X1 that represents the first communica- 
tion terminal device user and symbol image Y2 that rep- 
resents the second communication terminal device user 
are displayed on the second communication terminal 
device as shown in Fig. 4B. While the symbol images 
are displayed on these communication terminal devices 
as shown in Figs. 4Aand 4B, physical quantity data and 
information data are transmitted and received. Informa- 
tion data is received by information data input unit 11 
and transferred to input data storage unit 16, When the 
communication terminal device is vibrated, swung in the 
vertical or horizontal directions, or gripped tightly by its 
user, physical quantity data is detected by physical de- 
tection unit 12. Each kind of physical quantity data is 
digitized and output by physical quantity data output unit 
13. When the first communication terminal device user 
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presses or strokes image X2 that represents the receiv- 
er of the data from the first communication terminal de- 
vice with pen 101 a', or when the first communication 
terminal device user moves the point of pen 101a' up 
and down while pressing the head of image Y1 that rep- s 
resents the first communication terminal device user 
himself/herself with pen 101 a', image manipulation data 
that includes one code out of codes "00" to "1 1 ■ is output. 
The output image manipulation data is stored in input 
data storage unit 1 6. 

Each piece of data stored in input data storage unit 

16 is controlled by transmission control unit 17 and 
transmitted from transmission unit 1 9. The process per- 
formed by transmission control unit 17 is shown in the 
flowcharts in Figs. 9 and 10. The following is the expla- 
nation of the process performed by transmission control 
unit 17 with reference to Figs. 9 and 10. 

Transmission control unit 1 7 judges whether data is 
stored in input data storage unit 16 (Step S101). When 
no data is stored in input data storage unit 16, transmis- 
sion control unit 17 waits for data to be stored in input 
data storage unit 16 (Step S124). When data is stored 
in input data storage unit 16, transmission control unit 

17 judges the contents of the data. 

When only information data is included in the data, 
transmission control unit 17 preferentially transmits any 
audio data that is present (Steps S1 03 and S104). When 
other kinds of data apart from audio data is included in 
the information data and the transmission of the data is 
indicated by transmission data instruction unit 18, trans- 
mission control unit 17 transmits the data (Steps S105 
and S106). The data that has been transmitted is then 
cleared from input data storage unit 16. This is also the 
case in the process described below. 

When only physical quantity data or image manip- 
ulation data is stored in input data storage unit 16, un- 
less transmission data instruction unit 18 indicates that 
the data transmission should be stopped, the data is 
transmitted (Steps S108 to S110). 

When information data, physical quantity data, and 
image manipulation data are stored in input data storage 
unit 16, transmission control unit 17 controls the trans- 
mission as described below. When the information data 
includes other kinds of data apart from audio data, and 
when transmission data instruction unit 18 indicates 
transmission, transmission control unit 17 preferentially 
transmits the information data (Steps S1 1 1 to S113). 

When the information data includes audio data, 
transmission control unit 17 judges whether moving 
speed data is included in the physical quantity data 
(Steps S114 and S115), and further judges whether 
code "10" or "11" is included in the image data, that is, 
whether there is image manipulation data that affects 
the audio data (Steps S116 and S119). 

When judging that moving speed data is not includ- 
ed in the physical quantity data and that code "10" or 
"11" is included in the image manipulation data, trans- 
mission control unit 17 transmits the image manipulation 



data that includes code "10" or "11 " first and then trans- 
mits the audio data (Step S1 1 7). When no moving speed 
data is included in the physical quantity data and neither 
of codes "10" and "11" is included in the image manip- 
ulation data, transmission control unit 17 transmits the 
audio data as normal (Step S118). 

When moving speed data is included in the physical 
quantity data and neither of codes "10" and "11" is in- 
cluded in the image manipulation data, transmission 
control unit 17 divides the physical quantity data into a 
plurality of parts to be inserted between the pieces of 
audio data, and transmits the physical quantity data and 
the audio data (Step S120). When moving speed data 
is included in the physical quantity data and code "10" 
or "1 1 " is included in the image manipulation data, trans- 
mission control unit 1 7 transmits the image manipulation 
data that includes code "10" or "11" first, then divides 
the physical quantity data into a plurality of parts to be 
inserted between the pieces of audio data, and trans- 
mits the physical quantity data and the audio data (Step 
S121). 

When physical quantity data or image manipulation 
data to be transmitted is further stored in input datastor- 
age unit 16, transmission control unit 17 transmits the 
physical quantity data or image manipulation data 
(Steps S122 and S123). Transmission control unit 17 
repeats the above-described process until the line be- 
tween these communication terminal devices is discon- 
nected (Step S124). The above-described process that 
is performed by transmission control unit 17 and shown 
in Figs. 9 and 10 is, however, merely one example. A 
variety of processes may be performed by transmission 
control unit 17 according to the circumstances. 

The following is the explanation of reception system 
2 included in the second communication terminal de- 
vice, or portable communication terminal device A2. 
First of all reception unit 21 receives data transmitted 
from the first communication terminal device, or portable 
communication terminal device Al, and output data 
judgement storage unit 22 judges the kind of data and 
stores the daLa. Then the information data included in 
the data that is stored in output data judgement storage 
unit 22 is output from information data output unit 26 
through data editing unit 25a as it is. 

On the other hand, the physical quantity data and 
the image manipulation data stored in output data judge- 
ment storage unit 22 is analyzed by analysis unit 23, 
and operation control unit 24 controls the operations 
performed by operation unit 25 according to the result 
of the analysis. The operations performed by operation 
control unit 24 are shown in the flowcharts in Figs 11, 
1 2, and 1 3. The following is a detailed explanation of the 
process performed by operation control unit 24 with ref- 
erence to the flowcharts in Figs 11,12, and 1 3. In these 
operations the physical quantity data and the image ma- 
nipulation data that has been analyzed by analysis unit 
23, and the information data that has been output from 
information data output unit 26 is cleared from output 
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data judgement storage unit 22. 

Operation control unit 24 judges whether the mode 
of operation unit 25 is set as "consecutive operation 
mode" (Step S201), and performs different operations 
when the mode of operation unit 25 is set as another 5 
mode. The explanation of the operations that are per- 
formed when the mode of operation unit 25 is set as the 
mode other than "consecutive operation mode" will be 
given below. 

When analysis unit 23 judges that the first commu- 
nication terminal device is vibrating' according to the 
analysis result, operation control unit 24 has data editing 
unit 25a add vibrato to audio data, drives drive unit 25b, 
and has image editing unit 25c vibrate the symbol image 
that represents the first communication terminal device 
user (Steps S202 and S203). As a result, when the first 
communication terminal device is vibrated, the receiving 
communication terminal device user finds that the voice 
of the first communication terminal device user emitted 
from loudspeaker 102b of the second communication 
terminal device vibrates, the second communication ter- 
minal device itself vibrates, and symbol image X1 that 
represents the user of the first communication terminal 
device and is shown on display 1 02a of the second com- 
munication terminal device vibrates as shown in Fig. 14. 

When analysis unit 23 judges that the first commu- 
nication terminal device is swinging in the horizontal di- 
rection, operation control unit 24 has image editing unit 
25c shake the head of the symbol image that represents 
the first communication terminal device user (Steps 
S204 and S205). As a result, when the first communi- 
cation terminal device user swings the first communica- 
tion terminal device in the horizontal direction, the re- 
ceiving communication terminal device user finds that 
symbol image X1 shown on display 1 02a of the second 
communication terminal device shakes its head as 
shown in Fig. 15. 

When analysis unit 23 judges that the first commu- 
nication terminal device is swinging in the vertical direc- 
tion, operation control unit 24 has image editing unit 25c 
nod the head of the symbol image that represents the 
first communication terminal device user (Steps S206 
and S207). As a result, when the first communication 
terminal device user swings the first communication ter- 
minal device in the vertical direction, the receiving com- 
munication terminal device user finds that symbol image 
X1 on display 1 02a of the second communication termi- 
nal device nods its head as shown in Fig. 16. 

When pressure data is stored in output data judge- 
ment storage unit 22, operation control unit 24 controls 
the amount of red light and green light emitted from the 
LEDs in illumination unit 25d according to the amount 
of the pressure data. More specifically, operation control 
unit 24 has the red LEDs in illumination unit 25d increase 
the emission amount as the amount of pressure data 
increases and has the green LEDs increase the emis- 
sion amount as the amount of pressure data decreases 
(Steps S208 and S209). As a result, the color of light 



that is emitted from LEDs 102d of the second commu- 
nication terminal device changes between red and 
green according to the amount of pressure applied to by 
the first communication terminal device when the user 
grips the first communication terminal device as shown 
in Fig. 17. 

When code "00" is detected as image manipulation 
data, that is, when analysis unit 23 judges that symbol 
image X2 that represents the second communication 
terminal device user is pressed with the point of pen 
101a' on the display of the first communication terminal 
device, operation control unit 24 has image editing unit 
25c show a painful expression on symbol image Y2 that 
represents the second communication terminal device 
user, and has sound effects unit 25e emit a shriek (Steps 
S210 and S211). As a result, the action performed by 
the first communication terminal device user is con- 
veyed to the second communication terminal device us- 
er in images and sound as shown in Fig. 18. 

When code "01 B is detected as image manipulation 
data, that is, when analysis unit 23 judges that symbol 
image X2 that represents the second communication 
terminal device user is stroked with the point of pen 
101a' on the display of the first communication terminal 
device, operation control unit 24 has image editing unit 
25c show an amused expression on symbol image Y2 
that represents the second communication terminal de- 
vice user, and has sound effects unit 25e emit a laugh 
(Steps S212 and S213). As a result, the image and the 
sound corresponding to the result of the action per- 
formed on the first communication terminal device are 
expressed by the second communication terminal de- 
vice as shown in Fig. 1 9. 

When code "10" is detected as image manipulation 
data, that is, when analysis unit 23 judges that the first 
communication terminal device user presses the head 
of image Y1 that represents the first communication ter- 
minal device user with the point of pen 1 01 a' and moves 
the point of the pen up, operation control unit 24 raises 
the tone of the voice transmitted to data editing unit 25a 
(Steps S21 4 and S21 5). As a result, the tone of the voice 
that is input in the first communication terminal device 
along with the above-described action is raised when 
the voice is output from the second communication ter- 
minal device as shown in Fig. 20. 

When code "11 " is detected as image manipulation 
data, that is, when analysis unit 23 judges that the first 
communication terminal device user presses the head 
of image Y1 that represents the first communication ter- 
minal device user with the point of pen 1 01 a' and moves 
the point of the pen down, operation control unit 24 low- 
ers the tone of the voice transmitted to data editing unit 
25a (Steps S216 and S217). As a result, the tone of the 
voice that is input in the first communication terminal de- 
vice is lowered when the voice is output from the second 
communication terminal device as shown in Fig. 21. 

The explanation of the operations that are per- 
formed when the mode of operation unit 25 is set as 
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"consecutive operation mode" will be given below. 
When analysis unit 23 analyzes that the first communi- 
cation terminal device is vibrating, or when code "01 ■ is 
detected as image manipulation data, operation control 
unit 24 changes the predetermined operations to be per- 
formed by operation unit 25 in the "consecutive opera- 
tion mode" (Steps S21B and S219). 

More specifically, the first communication terminal 
device user sets the mode of operation unit 25 in the 
second communication terminal device as "consecutive 
operation mode", has operation unit 25 repeat the op- 
erations in which the red LEDs flash once, one handclap 
is emitted, the green LEDs flash once, and two hand- 
claps are emitted, and has the operation unit 25 display 
an video in which an image dances to the rhythm of the 
flashes and the handclaps on display 1 02a in the second 
communication terminal device as shown in Fig. 22A. 
When the first communication terminal device is vibrat- 
ed, or when symbol image X2 that represents the sec- 
ond communication terminal device user shown on dis- 
play 102a in the first communication terminal device is 
stroked with pen 101a', the operations repeated by op- 
eration unit 25 in the second communication terminal 
device changes as shown in Fig. 22B. More specifically, 
the red LEDs and the green LEDs are illuminated alter- 
nately, the handclaps are successively emitted, and the 
image in the video on display 1 02a dances more actively 
to the rhythm of the flashes and the handclaps. 

Operation control unit 24 repeats the above-de- 
scribed process until the line between these communi- 
cation terminal devices is disconnected (Step S220). In 
the above-described communication between those two 
communication terminal devices, not only information 
data but also the emotions of the first communication 
terminal device user may be conveyed to the second 
communication terminal device user. 

While the first communication terminal device trans- 
mits the moving speed data in vibration, reciprocating 
motion in the horizontal direction, and reciprocating mo- 
tion in the vertical direction to the second communica- 
tion terminal device without classifying the kinds of 
movement, and the second communication terminal de- 
vice analyzes the moving speed data and has operation 
control unit 24 control operations according to the kind 
of the motion in the present embodiment, it is possible 
for a sending communication terminal device to classify 
the moving speed data according to the kind of motion. 

For instance, a code may be given to each kind of 
motion performed by the first communication terminal 
device according to the moving speed data calculated 
by physical quantity data output unit 1 3. When the mov- 
ing speed data calculated by physical quantity data out- 
put unit 1 3 indicates that the first communication termi- 
nal device performs reciprocating motion and the cycle 
of the motion is shorter than the predetermined period 
of time, the motion of the first communication terminal 
device is judged as vibration and code "00" is given to 
the motion. When the moving speed data indicates that 



the first communication terminal device performs recip- 
rocating motion in the horizontal direction and the cycle 
of the motion is equal to or longer than the predeter- 
mined period of time, the motion of the first communi- 

s cation terminal device is judged as swinging in the hor- 
izontal direction and code "01" is given to the motion. 
When the moving speed data indicates that the first 
communication terminal device performs reciprocating 
motion in the vertical direction and the cycle of the mo- 

10 tion is equal to or longer than the predetermined period 
of time, the motion of the first communication terminal 
device is judged as swinging in the vertical direction and 
code "10" is given to the motion. Then header that indi- 
cates that the data is physical quantity data is added to 

15 the moving speed data, and the moving speed data is 
output as physical quantity data in the same way as im- 
age manipulation data. Transmission control unit 17 
may exert transmission control on the moving speed da- 
ta according to the code. 

20 While operation control unit 24 prepares the oper- 
ations to be performed by operation unit 25 on physical 
quantity data and image manipulation data in advance 
in the present embodiment, the instructions as to the op- 
erations to be performed on physical quantity data and 

25 image manipulation data and the program that realizes 
these actions may be transmitted along with the physical 
quantity data and the image manipulation data from the 
first communication terminal device to be used by oper- 
ation control unit 24. 

30 Fig. 23 is a functional block diagram of transmission 
system 1a and reception system 2a in a communication 
terminal device according to another embodiment of the 
present invention. Transmission system 1a differs from 
transmission system 1 according to the above-de- 

35 scribed embodiment in including link information addi- 
tion unit 10a instead of transmission control unit 17 and 
including transmission unit 1 9a whose functions differ 
from those of transmission unit 19. Reception system 
2a differs from reception system 2 according to the 

40 above-described embodiment in including related infor- 
mation extraction unit 27. 

Link information addition unit 10a in transmission 
system 1a adds link information, which relates certain 
information data with certain physical quantity data or 

45 certain image manipulation data, to the certain informa- 
tion data and the certain physical quantity data or certain 
image manipulation data. More specifically, moving 
speed data is used as the certain physical quantity data, 
image manipulation data to which code '10" or "11" is 

so added is used as the certain image manipulation data, 
and audio data that is input while the certain physical 
quantity data and the certain image manipulation data 
are input is used as the certain information data. Each 
of the certain physical quantity data, the certain image 

55 manipulation data, and the certain information data is 
treated as one piece of information. The same code is 
added as the link information to the header of the certain 
physical quantity data and that of the certain information 
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data, or to the header of the certain image manipulation 
data and that of the certain information data. The period 
of time in which certain image manipulation data is input 
is the period of time from the start of an action to the 
issuing of the reset signal. s 

Transmission unit 19a transmits information data 
preferentially when transmitting the data that is stored 
in input data storage unit 16, giving top priority to the 
instructions from transmission data instruction unit 18. 

Related information extraction unit 27 in reception 
system 2a extracts the certain information data, and the 
certain physical quantity data or the certain image ma- 
nipulation data according to the link information. More 
specifically, the related audio data, and moving speed 
data or image manipulation data to which code "10" or 
"11 ' is added are extracted. 

The operations related to the components that are 
included only in above-described transmission system 
1a or reception system 2a, or whose functions differ 
from those of their counterparts in transmission system 
1 and reception system 2 will be explained below. Fig. 
24 is the flowchart that shows the operations. 

The first communication terminal device user inputs 
audio data when vibrating the first communication ter- 
minal device or when increasing or decreasing the 
height of the symbol image of the first communication 
terminal device user, and stores the physical quantity 
data or the image manipulation data, and the informa- 
tion data that is obtained from these actions in input data 
storage unit 16 (Step S301). 

Link information addition unit 10a adds link informa- 
tion to the physical quantity data or the image manipu- 
lation data, and the information data that are related to 
each other (Step S302). Transmission unit 19a trans- 
mits the data stored in input data storage unit 1 6 in order 
of storage, preferentially transmitting the information da- 
ta (Step S303). 

Reception unit 21 in the second communication ter- 
minal device receives the transmitted data, and stores 
the data in output data storage unit 22. Related informa- 
tion extraction unit 27 extracts the physical quantity data 
or the image manipulation data, and the audio data to 
which each the same link information has been added 
(Step S304). 

When the audio data to which the link information 
is added is stored in output data storage unit 22 and the 
physical quantity data or the image manipulation data 
to which the corresponding link information is added is 
not stored in output data storage unit 22, the audio data 
is not output from information data output unit 26 and 
will have been stored in output data storage unit 22 or 
in related information extraction unit 27 until the physical 
quantity data or the image manipulation data is stored 
in output data storage unit 22. 

Analysis unit 23 analyzes the physical quantity data 
or the image manipulation data to which the link infor- 
mation is added, and operation control unit 24 has in- 
formation data editing unit 25a edit the audio data that 



is extracted by related information extraction unit 27 ac- 
cording to the analysis result (Step S305). 

As described above, using link information to relate 
the physical quantity data or the image manipulation da- 
ta with the information data which is edited by informa- 
tion data editing unit 25a may enable data editing unit 
25a perform its operations properly in the construction 
shown in Fig. 23. 

The following is the explanation of another embod- 
iment of the present invention. 

Fig. 25 is the block diagram of transmission system 
1b. Transmission system 1b includes audio data con- 
version unit 10b that converts the audio data included 
in information data, and audio data conversion data out- 
put unit 1 0c that outputs the information about the audio 
data conversion as audio data conversion data accord- 
ing to the conversion performed by audio data conver- 
sion unit 10b. 

Audio data conversion unit 1 0b changes the tone of 
input audio data by changing the waveform of the input 
audio data. In the present embodiment the tone of audio 
data is raised and lowered by changing the frequency 
of the waveform. 

Audio data conversion data output unit 1 0c converts 
the amount of conversion performed on audio data into 
numerical value data within a fixed range, adds a piece 
of header that indicates that the data is audio data con- 
version data to each piece of the numerical value data, 
and outputs the data as audio data conversion data. Au- 
dio data conversion data transmission is dealt with by 
control unit 17 in the same manner as image manipula- 
tion data to which code "00" or "01 " is added. 

In the reception system that receives data from the 
transmission system whose construction has been ex- 
plained above, analysis unit 23 and operation control 
unit 24 handle the audio data conversion data. More 
specifically, analysis unit 23 analyzes the audio data 
conversion data and extracts the numerical value data 
that represents the amount of conversion. Operation 
control unit 24 has operation unit 25 perform operations 
corresponding to the analysis result. For instance, when 
the tone of audio data is raised by audio data conversion 
unit 10b, operation control unit 24 has operation unit 25 
convert symbol image X1 that represents the sending 
communication terminal device user to a woman ac- 
cording to the change of the tone of voice. When the 
tone of voice data is lowered, operation control unit 24 
has operation unit 25 convert symbol image X1 to a 
man. 

As a result, when audio data conversion data is 
transmitted and received, it is possible to have the re- 
ception system perform operations according to the ac- 
tion performed by the sending communication terminal 
device user in the same manner as image manipulation 
data is transmitted and received. 

Other possible modifications are given below. 

(1 )While each of the communication terminal devic- 
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es shows the symbol images of the communication 
terminal device user and its receiving communica- 
tion terminal device user on the display in the above 
embodiments, only the symbol image of the receiv- 
ing communication terminal device user may be 
shown on the display. 

(2) A photograph image of a user may be used in- 
stead of a symbol image. In case of a photograph 
image, however, the amount of information to be 
transmitted is larger, so that it will take longer to 
transmit the image data. Text data may be also used 
instead of a symbol image. In case of text data, the 
actions peculiar to a symbol image such as nodding 
and shaking the head are not shown on the display, 
and actions such as tearing and crumpling the text 
data, and turning the page of the text data may be 
shown instead. 

(3) While three-dimensional image data is used to 
display a symbol image in the above-described em- 
bodiments, animation data that includes all images 
necessary for the display may be used instead. 
While the symbol image of a communication termi- 
nal device user is transmitted to the other commu- 
nication terminal device at the beginning of commu- 
nication in the above-described embodiments, the 
symbol image may be transmitted every time an ac- 
tion is performed by the user. 

(4) While link information is added to physical quan- 
tity data and to the data to be processed with the 
physical quantity data, or an identifier is added to 
physical quantity data to indicate the data to be 
processed with the physical quantity data in the 
above-described embodiments, if physical quantity 
data and the data to be processed with the physical 
quantity data are transmitted in order of time, it is 
not necessary to add link information nor identifier 
to the data. 

(5) while the vibration or the swing of a communica- 
tion terminal device is detected as physical quantity 
data in the above-described embodiments, when a 
mouse, or a touch pen is attached to the communi- 
cation terminal device, the vibration or the swing of 
the mouse or the touch pen may be detected as 
physical quantity data. 

Although the present invention has been fully de- 
scribed by way of examples with reference to the ac- 
companying drawings, it is to be noted that various 
changes and modifications will be apparent to those 
skilled in the art. Therefore, unless such changes and 
modifications depart from the scope of the present in- 
vention, they should by construed as being included 
therein. 



Claims 

1. A sending communication terminal device that 



transmits data to a receiving communication termi- 
nal device, comprising: 

a detection means for detecting physical quan- 
5 tities that affect one of a main body of the send- 

ing communication terminal device and a part 
of the sending communication terminal device; 
a preparation means for preparing one of target 
data that is to be processed with the physical 
?0 quantities detected by the detection means and 

an identifier for identifying the target data; and 
a transmission means for transmitting the phys- 
ical quantities and one of the target data and 
the identifier prepared by the preparation 
*5 means in a form that enables the physical quan- 

tities and one of the target data and the identi- 
fier to be identified. 

2. The sending communication terminal device ac- 
20 cording to Claim 1 , wherein 

the preparation means also prepares at least 
one of audio data, text data, and image data, 
and 

25 the transmission means additionally transmits 

at least the one of the audio data, text data, and 
the image data prepared by the preparation 
means in a form that enables each of the audio 
data, the text data, and the image data to be 

30 identified. 

3. The sending communication terminal device ac- 
cording to Claim 2, wherein 

the transmission means includes a link infor- 
ms mat ion addition means for adding link information, 
which relates the physical quantities with the target 
data, to at least the physical quantities. 

4. The sending communication terminal device ac- 
40 cording to Claim 3, wherein the detection means in- 
cludes: 

an action detection sensor for detecting an ac- 
tion that is performed by a user of the sending 

45 communication terminal device on one of a 

main body of the sending communication ter- 
minal device and a part of the sending commu- 
nication terminal device; and 
a physical quantity data output unit for creating 

so data relating to an action type and a direction 

using signals detected by the action detection 
sensor and for outputting the created data. 

5. The sending communication terminal device ac- 
55 cording to Claim 4, wherein 

the action detection sensor is one of an accel- 
eration sensor and a speed sensor that is installed 
in one of the main body of the sending communica- 
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cording to Claim 10, wherein 

the symbol image data is three<Jimensional im- 
age data, and 

5 the transmission means performs a process by 

which the three-dimensional image data is 
transmitted at a beginning of communication 
with the receiving communication terminal de- 
vice. 

10 

12. The sending communication terminal device ac- 
cording to Claim 1 , wherein 

the target data is symbol image data that met- 
15 aphorically represents a user of the sending 

communication terminal device, 
the receiving sending communication terminal 
device stores the symbol image data that met- 
aphorically represents the user of the sending 
20 communication terminal device in advance, 

the preparation means includes a storage 
means for storing the identifier that identifies 
the symbol image data of the sending commu- 
nication terminal device user, and 
25 the transmission means transmits the identifier 

at a beginning of communication with the re- 
ceiving sending communication terminal de- 
vice. 



tion terminal device and the part of the sending 
communication terminal device, and detects wheth- 
er one of the main body of the sending communica- 
tion terminal device and the part of the sending 
communication terminal device is being vibrated or 
swung, 

6. The sending communication terminal device ac- 
cording to Claim 1 further comprising a display for 
displaying an image that is the target data that is to 
be processed with the physical quantities, wherein 

the detection means includes a touch panel 
that is provided on the display, and a processing cir- 
cuit that detects an action performed by the user of 
the sending communication terminal device on the 
touch panel and interprets the detected action per- 
formed by the user as one of pressing, stroking, and 
rubbing. 

7. The sending communication terminal device ac- 
cording to Claim 6, wherein 

the transmission means includes a link infor- 
mation addition means for adding link information, 
which relates the physical quantities with the target 
data, to at least the physical quantities. 

8. The sending communication terminal device ac- 
cording to Claim 7, wherein the detection means in- 
cludes: 

an action detection sensor for detecting an ac- 
tion that is performed by a user of the sending 
communication terminal device on one of a 
main body of the sending communication ter- 
minal device and a part of the sending commu- 
nication terminal device; and 
a physical quantity data output unit for creating 
data relating to an action type and a direction 
using signals detected by the action detection 
sensor and for outputting the created data. 

9. The sending communication terminal device ac- 
cording to Claim 1 , wherein 

the target data is symbol image data that met- 
aphorically represents a user of the sending 
communication terminal device, and 
the preparation means includes a storage 
means for storing the symbol image data. 

10. The sending communication terminal device ac- 
cording to Claim 9, wherein 

the transmission means includes a link infor- 
mation addition means for adding link information, 
which relates the physical quantities with the target 
data, to at least the physical quantities. 

11. The sending communication terminal device ac- 



30 13. The sending communication terminal device ac- 
cording to Claim 1 2, wherein the symbol image data 
of the sending communication terminal device user 
is animation data. 

35 14. The sending communication terminal device ac- 
cording to Claim 1, wherein 



the target data is audio data for a voice of a user 
of the sending communication terminal device, 
40 the preparation means is a microphone which 

detects the voice of the user of the sending 
communication terminal device, and 
the transmission means transmits the audio da- 
ta in real time. 

45 

15. A receiving communication terminal device that re- 
ceives data transmitted from a sending communi- 
cation terminal device, comprising: 

50 a reception means for receiving the data trans- 

mitted from the sending communication termi- 
nal device; 

a division means for grouping the data trans- 
mitted from the sending communication termi- 
55 nal device into a first data that represents phys- 

ical quantities detected by the sending commu- 
nication terminal device and a second data that 
is to be processed with the physical quantities; 
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a processing means for processing the second 
data according to the first data; and 
an expressing means for expressing for a user 
of the receiving communication terminal device 
the second data after the processing by the 
processing means. 

16. The receiving communication terminal device ac- 
cording to the Claim 15, wherein 

the division means includes a data type judge- 
ment unit for detecting identifiers that are added to 
the data transmitted from the sending communica- 
tion terminal device, and for distinguishing the first 
data and the second data based on the detected 
identifiers. 

17. The receiving communication terminal device ac- 
cording to the Claim 16, wherein the processing 
means includes: 

an analysis unit for analyzing the first data that 
affects the sending communication terminal de- 
vice to detect a type of the physical quantities; 
and 

a processing unit for processing the second da- 
ta according to an analysis result of the analysis 
unit. 

18. The receiving communication terminal device ac- 
cording to the Claim 17, wherein 

the second data obtained from the grouping by 
the division means includes audio data and im- 
age data, 

the audio data is voice of a user of the sending 
communication terminal device, 
the image data is symbol image data that met- 
aphorically represents the user of the sending 
communication terminal device, 
the analysis unit analyzes the first data to de- 
termine that the physical quantities that affect 
the sending communication terminal device are 
for a vibration and or a swing, 
the processing unit refers to an analysis result 
of the analysis unit, adds vibrato to the audio 
data, and shakes a head of the symbol image 
data of the sending communication terminal de- 
vice user in one of vertical and horizontal direc- 
tions, according to the result and 
the expressing means includes a loudspeaker 
that emits the audio data after processing by 
the processing unit and a display that displays 
the image data after processing by the process- 
ing unit. 

19. The receiving communication terminal device ac- 
cording to the Claim 18, wherein 



the symbol image data of the sending commu- 
nication terminal device user is three-dimen- 
sional image data, 
the processing unit includes: 

5 

an image data creation unit for creating 
variations of the image data seen from a 
plurality of camera angles using the three- 
dimensional image data; and 
w a selection unit for selectively outputting 

the variations of the image data one at a 
time to the display according to the analy- 
sis result of the analysis unit. 

*5 20. The receiving communication terminal device ac- 
cording to the Claim 18 further comprising an oscil- 
lation unit, wherein 

when the physical quantities that affect the 
20 sending communication terminal device are for 

a swing, the oscillation unit is driven and swings 
the receiving communication terminal device, 
and 

when the physical quantities that affect the 
25 sending communication terminal device are for 

a vibration, the oscillation unit is driven and vi- 
brates the receiving communication terminal 
device. 

30 21. A receiving communication terminal device that re- 
ceives data transmitted from a sending communi- 
cation terminal device comprising: 

a reception means for receiving the data trans- 
35 mitted from the sending communication termi- 

nal device; 

a data type judgement means for detecting 
identifiers that are added to the data transmit- 
ted from the sending communication terminal 
40 device, and for distinguishing first data that rep- 

resents physical quantities and second data 
that is other data, based on the detected iden- 
tifiers; 

a to-be-processed data extraction means for 
45 detecting link information included in at least 

one of the first data and the second data, and 
extracting data, from the second data, which is 
related to the first data by the link information; 
a processing means for processing the extract- 
50 ed data with the first data; and 

an expressing means for expressing for a user 
of the receiving communication terminal device 
the extracted data after processing by the 
processing means. 

55 

22. The receiving communication terminal device ac- 
cording to the Claim 21, wherein the processing 
means includes: 
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an analysis unit for analyzing the first data that 
affects the sending communication terminal de- 
vice to detect a type of the physical quantities; 
and 

a processing unit for processing the second da- 
ta according to an analysis result of the analysis 
unit. 

23. The receiving communication terminal device ac- 
cording to the Claim 22, wherein 

the extracted data includes audio data and im- 
age data, 

the audio data is voice of a user of the sending 
communication terminal device, 
the image data is symbol image data that met- 
aphorically represents the user of the sending 
communication terminal device, 
the analysis unit analyzes the first data to de- 
termine that the physical quantities that affect 
the sending communication terminal device are 
for a vibration and or a swing, 
the processing unit refers to an analysis result 
of the analysis unit, adds vibrato to the audio 
data, and shakes a head of the symbol image 
data of the sending communication terminal de- 
vice user in one of vertical and horizontal direc- 
tions, according to the result and 
the expressing means includes a loudspeaker 
that emits the audio data after processing by 
the processing unit and a display that displays 
the image data after processing by the process- 
ing unit. 

24. A receiving communication terminal device that re- 
ceives data transmitted from a sending communi- 
cation terminal device comprising: 

a reception means for receiving the data trans- 
mitted from the sending communication termi- 
nal device; 

a division means for grouping the data trans- 
mitted from the sending communication termi- 
nal device into a first data that represents phys- 
ical quantities detected by the sending commu- 
nication terminal device and a second data that 
specifies data that is to be processed with the 
physical quantities; 

a storage means for storing the data to be proc- 
essed; 

a processing means for extracting the data that 
is to be processed and is specified by the sec- 
ond data, and for processing the extracted data 
according to the first data; and 
an expressing means for expressing for a user 
of the receiving communication terminal device 
the extracted data after the processing by the 
processing means. 



25. The receiving communication terminal device ac- 
cording to the Claim 24, wherein the processing 
means comprises: 

5 an analysis unit for analyzing the first data to 

detect a type of the physical quantities that af- 
fect the sending communication terminal de- 
vice; and 

a processing unit for processing the data that 
'0 is to be processed and is specified by the sec- 

ond data according to an analysis result of the 
analysis unit. 

26. The receiving communication terminal device ac- 
15 cording to the Claim 25, wherein 

the data that is to be processed and is specified 
by the second data includes a symbol image 
data that metaphorically represents a user of 
20 the sending communication terminal device, 

the analysis unit analyzes the first data to de- 
termine that the physical quantities that affect 
the sending communication terminal device are 
for a vibration and a swing, 
25 the processing unit refers to an analysis result 

of the analysis unit, and shakes a head of the 
symbol image data of the sending communica- 
tion terminal device user in one of a vertical and 
a horizontal directions, and 
30 the expressing means is a display for displaying 

the symbol image data of the sending commu- 
nication terminal device user after processing 
by the processing unit. 

35 27. The receiving communication terminal device ac- 
cording to the Claim 24, wherein the processing 
means includes: 

an analysis unit for analyzing the first data to 
40 determine whether an image manipulation per- 

formed on the symbol image data that meta- 
phorically represents the user of the receiving 
communication terminal device is pressing or 
stroking; and 

45 a processing unit for processing the data that 

is to be processed and is specified by the sec- 
ond data according to an analysis result of the 
analysis unit. 

50 28. The receiving communication terminal device ac- 
cording to the Claim 27, wherein 

the data that is to be processed and is specified 
by the second data is symbol image data that 
55 metaphorically represents the user of the re- 

ceiving communication terminal device, 
the processing unit processes the symbol im- 
age data that metaphorically represents the us- 
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er of the receiving communication terminal de- 
vice, according to a type of the image manipu- 
lation that is an analysis result of the analysis 
unit, so that the symbol image data chanqes an 
expression to reflect the image manipulation, s 
and 

the expressing means is a display for displaying 
the symbol image data of the receiving commu- 
nication device user after the processing by the 
processing unit 10 

29. The receiving communication terminal device ac- 
cording to the Claim 28, wherein 

the symbol image data of the receiving commu- * 5 
nication terminal device user is three-dimen- 
sional image data, 
the processing unit includes: 

an image data creation unit for creating 20 
variations of image data seen from a plu- 
rality of camera angles using the three-di- 
mensional image data; and 
a selection unit for selectively outputting 
the variations of the image data one at a 25 
time to the display according to the analy- 
sis result of the analysis unit. 

30. The receiving communication terminal device ac- 
cording to the Claim 28, wherein 30 



the symbol image data of the receiving commu- 
nication terminal device user is animation data, 
and 

the processing unit selects predetermined ani- 
mation data according to the analysis result of 
the analysis unit, and outputs the predeter- 
mined animation data to the display. 

31. A communication terminal device that performs 
two-way communication with another communica- 
tion terminal device comprising; 

a display; 

a physical quantity detection means for detect- 
ing an action performed on a first image that is 
displayed on the display to create first opera- 
tion data, and for detecting an action performed 
on a second image that is displayed on the dis- 
play to create second operation data; 
a preparation means for preparing at least one 
of image data and audio data corresponding to 
at least one of the first and the second images; 
a transmission means for transmitting the first 
operation data, the second operation data, and 
a preparation data in a form which enables 
each of the first operation data, the second op- 
eration data, and the preparation data to be 



identified; 

a reception means for receiving data transmit- 
ted from the other communication terminal de- 
vice; 

a data judgement means forjudging that the da- 
ta transmitted from the other communication 
terminal device is one of received first opera- 
tion data, received second operation data, and 
received preparation data; 
a first processing means for processing the re- 
ceived preparation data that is received by the 
reception means according to the received first 
operation data; 

a second processing means for processing the 
received preparation data that is received by 
the reception means according to the received 
second operation data; 

an activation means for selectively activating 
one of the first processing means and the sec- 
ond processing means according to a result ob- 
tained by the data type judgement means; 
a display control means for having, when the 
preparation data that is processed by one of the 
first processing means and the second 
processing means is image data, a correspond- 
ing image on the display replaced with contents 
of the preparation data; and 
a sound emission means for emitting the con- 
tents of the preparation data when the prepa- 
ration data that is processed by one of the driv- 
en first processing means and the second 
processing means is audio data. 

32. The communication terminal device according to 
35 Claim 31, wherein 

the first image represents a user of the commu- 
nication terminal device, and 
the second image represents a user of the other 
40 communication terminal device. 

33. The communication terminal device according to 
Claim 32, wherein the image data prepared by the 
preparation means is three-dimensional image data 

45 of a symbol image that metaphorically represents 
the user of the communication terminal device. 

34. The communication terminal device according to 
Claim 33, wherein the transmission means includes 

so a link information addition unit for adding an identi- 
fier, which identifies the preparation data to be proc- 
essed, to one of the first operation data and the sec- 
ond operation data. 

55 35. The communication terminal device according to 
Claim 34, wherein the first processing means and 
the second processing means include: 
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an image editing unit for creating, when the 
preparation data that is received by the recep- 
tion means is three-dimensional image data, 
variations of the image data seen from a plural- 
ity of camera angles, using the three-dimen- & 
sional image data; and 

a selection unit for selecting image data seen 
from a predetermined camera angle out of the 
variations of the image data according to con- 
tents of one of the first and the second opera- 10 
tion data received by the reception means. 



36. The communication terminal device according to 
Claim 35, wherein 

the image editing unit includes an expression 
creation unit for partially processing image data 
seen from a predetermined camera angle to 
create a variation of the symbol image with a 
different expression, and 
wherein the selection unit selects a variation of 
the symbol image with the different expression 
according to contents of one of the received 
first and second operation data. 

37. A communication system in which two communica- 
tion terminal devices transmit and receive data 
comprising: 

a first communication terminal device that in- 
cludes: 



terminal device into a first data that repre- 
sents physical quantities detected by the 
first communication terminal device and a 
second data that is to be processed with 
the physical quantities; 
a processing means for processing the 
second data according to the first data; and 
an expressing means for expressing for a 
user of the second communication terminal 
device the second data after the process- 
ing by the processing means. 

38. The communication system according to Claim 37, 
wherein 

75 

the transmission means includes a link informa- 
tion addition means for adding link information, 
which relates the physical quantities with the 
target data, to at least the physical quantities, 

20 and 

the division means includes a data type judge- 
ment unit for detecting identifiers that are add- 
ed to the data transmitted from the first commu- 
nication terminal device, and for distinguishing 

2$ the first data and the second data based on the 

detected identifiers. 

39. The communication system according to Claim 38, 
wherein 

30 

the detection means includes: 

an action detection sensor for detecting an 
action that is performed by a user of the first 
communication terminal device on one of 
the main body of the first communication 
terminal device and the part of the first 
communication terminal device; and 
a physical quantity data output unit for cre- 
ating data relating to an action type and a 
direction using signals detected by the ac- 
tion detection sensor and for outpu tting the 
created data, and 

wherein the processing means includes: 

an analysis unit for analyzing the first data 
that affects the first communication termi- 
nal device to detect a type of the physical 
quantities; and 

a processing unit for processing the sec- 
ond data according to an analysis result of 
the analysis unit. 

40. The communication system according to Claim 39, 
wherein 

the second data obtained from the grouping by 



a detection means for detecting physical 
quantities that affect one of a main body of 
the first communication terminal device 35 
and a part of the first communication termi- 
nal device; 

a preparation means for preparing one of 
target data that is to be processed with the 
physical quantities detected by the detec- 40 
tion means and an identifier for identifying 
the target data; and 

a transmission means for transmitting the 
physical quantities and one of the target 
data and the identifier prepared by the 45 
preparation means in a form that enables 
the physical quantities and one of the tar- 
get data and the identifier to be identified; 
and 

so 

a second communication terminal device that 
includes: 

a reception means for receiving data trans- 
mitted from the first communication termi- 55 
nal device; 

a division means for grouping the data 
transmitted from the first communication 
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the division means includes audio data and im- 
age data, 

the audio data is voice of the user of the first 
communication terminal device, 
the image data is symbol image data that met- s 
aphorically represents the user of the first com- 
munication terminal device, 
the analysis unit analyzes the first data to de- 
termine that the physical quantities that affect 
the first communication terminal device are for io 
a vibration and or a swing, 
the processing unit refers to an analysis result 
of the analysis unit, adds vibrato to the audio 
data, and shakes a head of the symbol image 
data of the first communication terminal device is 
user in one of vertical and horizontal directions, 
according to the result, and 
the expressing means includes a loudspeaker 
that emits the audio data after processing by 
the processing unit and a display that displays 20 
the image data after processing by the process- 
ing unit. 
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